Skip to content

refactor(compute): decouple gateway driver composition - #2823

Merged
drew merged 9 commits into
mainfrom
refactor/2417-server-driver-decoupling/drew
Sep 1, 2026
Merged

refactor(compute): decouple gateway driver composition#2823
drew merged 9 commits into
mainfrom
refactor/2417-server-driver-decoupling/drew

Conversation

@drew

@drew drew commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Move first-party compute-driver composition and VM process ownership into openshell-gateway, leaving openshell-server independent of compute backend implementations. The standard gateway installs drivers through a generic registry, can build without linked compute backends, and preserves the public compute-driver RPC path for external drivers.

Related Issue

Part of #2417.

Changes

  • add openshell-gateway as the binary and first-party compute-driver composition crate
  • gate linked first-party registrations behind the in-tree-compute-drivers feature
  • replace backend-specific server construction with generic driver registrations, factories, metadata, and build context
  • move VM subprocess ownership and first-party Docker, Podman, Kubernetes, VM, and Windows MXC composition to the gateway boundary
  • preserve driver-free gateway builds and external-driver operation for Docker, Podman, Kubernetes, and VM
  • move Docker and Podman socket discovery into their driver crates, including native and machine-backed Podman discovery
  • keep compute-driver telemetry categories bounded and move driver-specific tracing setup to registrations
  • carry user_namespaces as a typed compute-driver field, translate it to Kubernetes hostUsers, and accept the legacy platform_config.host_users encoding during rolling upgrades
  • carry the canonical create-time SandboxPolicy in DriverSandboxSpec.policy, so validation and creation use one complete sandbox message
  • use the default-safe driver_reports_runtime_readiness capability solely for readiness composition instead of coupling readiness and policy behavior in a runtime-control enum
  • leave later policy revisions on the existing GetSandboxConfig and ReportPolicyStatus extension contract rather than adding a compute-driver reconcile RPC or recreate capability
  • compose MXC through the ordinary InProcess variant, with driver-reported readiness and create-time policy enforcement; remove the MXC-only policy side channel and InProcessWithSandboxPolicy
  • remove the gateway-side live-policy rejection that was tied to backend identity
  • update architecture, public compute-driver documentation, packaging, deployment scripts, E2E wiring, Windows build guidance, and related agent skills
  • remove stale backend-specific comments and obsolete openshell-server binary build references

Testing

  • mise run pre-commit
  • OPENSHELL_SYSTEM_GATEWAY_DIR=/tmp/codex-openshell-empty-system-config mise run test
  • focused server test suite and Linux MXC package test target (cargo test -p openshell-server -p openshell-driver-mxc)
  • cargo check -p openshell-server -p openshell-gateway --all-features
  • mise run helm:test (106 tests)
  • cargo check -p openshell-gateway --no-default-features
  • cargo check -p openshell-gateway --features in-tree-compute-drivers
  • focused gateway registration and Kubernetes user-namespace tests
  • mise run e2e:gateway:no-compute-drivers
  • Docker, Podman, and Kubernetes external-driver E2E jobs
  • Linux and macOS gateway build jobs
  • remaining branch E2E and aggregate CI jobs are still running
  • native Windows MSVC build/test lane (not available on this host)

Checklist

@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from 6d719f2 to 96f603d Compare August 20, 2026 05:10
@drew

drew commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test

@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from 96f603d to bb88ad5 Compare August 20, 2026 14:28
@drew

drew commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test

@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from bb88ad5 to d3ae889 Compare August 20, 2026 14:32
@drew

drew commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test

@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from d3ae889 to 69ddee1 Compare August 20, 2026 14:59
@drew

drew commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test

@elezar
elezar force-pushed the refactor/2417-server-driver-decoupling/drew branch from 69ddee1 to 00eff3d Compare August 20, 2026 15:35
@elezar

elezar commented Aug 20, 2026

Copy link
Copy Markdown
Member

@drew This branch update is rebase-only: PR #2823 was rebased onto the restored PR #2822 head. No intentional feature or scope changes were added to this PR; the conflict resolution preserves the parent stack behavior while retaining the existing gateway-composition and telemetry commits.

@elezar
elezar force-pushed the refactor/2417-server-driver-decoupling/drew branch from 00eff3d to a8047fd Compare August 20, 2026 15:44
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from a8047fd to 4348c48 Compare August 20, 2026 16:10
@copy-pr-bot

copy-pr-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from 4348c48 to d910ae3 Compare August 20, 2026 19:18
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from d910ae3 to 31a9534 Compare August 20, 2026 20:50
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from 31a9534 to 507d4c3 Compare August 20, 2026 21:56
@drew
drew requested a review from a team as a code owner August 31, 2026 18:01
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from d9f9e27 to 1cf8a8b Compare August 31, 2026 19:11
@drew drew added the test:e2e Requires end-to-end coverage label Aug 31, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 1cf8a8b. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from aa013d6 to 6ad2cb0 Compare September 1, 2026 05:47
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch 4 times, most recently from 09ce788 to 59d91c8 Compare September 1, 2026 17:32
Comment thread crates/openshell-server/src/cli.rs
pimlock
pimlock previously approved these changes Sep 1, 2026
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from 59d91c8 to c587207 Compare September 1, 2026 20:06
drew and others added 9 commits September 1, 2026 13:22
Move first-party composition and VM process ownership into openshell-gateway, leaving openshell-server backend-independent. Update packaging and build references with the new crate, simplify the compiled-driver boundary, and keep the driver-free gateway path buildable with bundled Z3 tooling.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
drew force-pushed the refactor/2417-server-driver-decoupling/drew branch from c587207 to ee76daf Compare September 1, 2026 20:24
@drew
drew added this pull request to the merge queue Sep 1, 2026
Merged via the queue into main with commit 9ca19e6 Sep 1, 2026
123 of 125 checks passed
@drew
drew deleted the refactor/2417-server-driver-decoupling/drew branch September 1, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants